1. Field of the Invention
This invention relates to the management of interfaces between high speed buses and memory. In particular, the invention relates to an arrangement of non-volatile integrated circuit memory, such as flash memory, that supports operation with a high speed bus.
2. Description of the Related Art
Large scale data storage systems are being used in an increasing variety of settings. Thus, flexibility in the design of the access systems used with these systems is becoming Ad increasingly important. One approach to improving the flexibility which has evolved is called a storage area network SAN. In the SAN environment, heterogeneous storage systems are being deployed which allow for greater flexibility in the use and management of data. In a SAN, the storage systems are interconnected by high-speed communication channels, such as the fiber channel networks. Thus, for the best performance, the interfaces to the memory systems in the SAN must be as fast as possible.
One kind of memory system which is not been widely applied to the SAN environment is non-volatile solid-state memory, such as memory systems using integrated circuit flash memory devices. One reason non-volatile solid state memory is not in wide-spread use arises from the relatively slow processes used for storing data in such devices. It is difficult for a system based on an array of flash memory integrated circuits, for example, to keep up with a high-speed communication channel feeding data.
The current generation of flash memory modules represented by devices such as the Toshiba TC58256FT, generally has a relatively long write period which varies in length over the life of the device from about 200 xcexcs to as much as 1000 xcexcs or more per write cycle. Read operations are much faster, but can still take 10 xcexcs or more. Furthermore, the memory modules have on chip buffers, which accept data bytes at a clock speed up to about 20 MHz for example. Standard bus speeds are generally much faster and carries eight bytes per cycle. For example, the PCI bus operates typically at 33 or 66 MHz and carry 64 bits or 8 bytes per cycle. This means that there cannot be a write to the flash memory module during each bus cycle.
In order to transfer data from a computer bus to flash memory, typically a buffer is used. The buffer is designed to be big enough to hold the data received over the bus as the flash memory write cycles occur. For a representative system using current generation flash memory modules, a 16 KB FIFO buffer is required at the interface between the flash device and a 66 MHz, 64 bit PCI bus. The buffers often require extra board space, and are easily overrun by large data transfer operations.
Thus, this configuration does not permit the flash memory to be used in a sustained transfer of large files at the same speed as the computer bus. Further, if a faster bus is used, the performance of the flash memory becomes progressively worse compared to the capacity of the bus.
Accordingly, what is needed is a method and apparatus for interfacing a high speed bus with a flash memory or other non-volatile solid state memory devices.
A memory system with an array of non-volatile solid state memory devices including an interface for a high speed bus is described, supporting continuous writes at the bus speed of very large blocks of data, without the possibility of buffer overrun during most conditions.
An apparatus comprises an memory bus, a plurality of interface buffers, an array of non-volatile storage units, such as flash memory devices, and an interconnect system supporting data transfer among the components. The array includes sets and subsets of non-volatile storage units, referred to herein for convenience as platters having multiple banks, banks having multiple columns, and columns having multiple storage units. In one example, the array includes two platters, eight banks per platter, four columns per bank, and eight storage units per column, for a total of 256 storage units. Of course other configurations fall within the present invention using different combinations of units per column, columns per bank, and banks per platter.
The non-volatile storage units each have an input buffer for storing a page of data, and an input port coupled to input pins on the unit and to the input buffer. The page size and the size of the input port can vary, but for example, a page is 256, 512 or 1024 bytes, and the input port can accept one or two 8-bit bytes per storage unit clock cycle.
In one embodiment supporting continuous writes, there are at least N interface buffers f (f=0 to Nxe2x88x921), the interface buffers having a depth of Z cycles, at least N columns c (copyright)=0 to Nxe2x88x921) in each of at least M banks b (b =0 to Mxe2x88x921), and the page buffers in the non-volatile memory units include storage for at least X addresses in a page (i=0 to Xxe2x88x921). Logic in the system employs a process supporting continuous writes comprising writing data to bank b, page address i, and column c in a given input cycle i+c+b+Z from the interface buffer f to column c, for f and c going from 0 to Nxe2x88x921, and then incrementing i, for i going from 0 to Xxe2x88x921, and then incrementing b for b going from 0 to Mxe2x88x921. Z in preferred implementations ranges from 1 to 16.
The memory speed at which the input buffer can accept data can vary. In the following example, a typical speed of 16.5 MHz is used. The non-volatile storage units take a certain write time to store the page of data from the input buffer into the memory. The sets of non-volatile storage units are each coupled to a corresponding interface buffer by a memory bus. The memory bus supplies data from the buffers to the inputs of the non-volatile storage units at the memory speed. The input bus is coupled to the interface buffers to supply them with data. The input bus speed is typically several times faster than the memory speed. For example, the input bus speed might be 66 MHz as compared to a memory speed of 16.5 MHz. The write time for flash memory devices includes a write wait time plus a setup time plus the time to write the number of bytes required. For a column of eight devices with one byte input ports, a bus eight bytes wide can supply data to be written in one storage unit cycle in the column. For a buffer of 512 bytes, 512 storage unit cycles are used to fill the input buffers of the column of devices. Thus, in 512 storage unit cycles, 4192 (4K) bytes are stored in the column to be written into the non-volatile memory. The total time, considering zero wait states, is one storage unit cycle for a command, three cycles for address, 512 cycles for data, and the memory wait time. Thus, this total time ranges, for example, from about 232.182 micro-seconds to 1032.182 micro-seconds, with the bus coupled to the input port busy for 32.182 micro-seconds.
With a 16.5 MHz storage unit clock, 4 interleaved columns are used in each bank to keep up with a 66 MHz PCI bus. This provides for storage of 16K bytes within each 32.182 micro-second per bank interval at the speed of the incoming PCI bus. At the end of the per bank interval, the system switches to the next bank on the platter. The number of banks on the platter is selected so that total write time of, for example, about 250 micro-seconds elapses before the system reverts to the first bank. Multiple platters can be coupled in parallel with logical memory addressing for added memory capacity or in a series to handle longer write times.
The number of non-volatile storage banks in each array is going to be at least as great as the memory write time multiplied by the memory speed divided by the page size. For example, if the memory speed is 16.5 MHz, the page size is 512 bytes and the memory write time is 200 xcexcs, at least seven banks must be provided. More can be provided and in one embodiment, eight banks are used with these clock speed and input buffer parameters.
In one embodiment, the system includes control logic for accepting burst data transfers over the input bus and storing the burst data in the non-volatile storage units.
In one embodiment, the system includes logic for selecting a starting page in the non-volatile storage units to store the data burst.
In one embodiment, the system includes control logic for providing a destination page and control information to the non-volatile storage units.
In one embodiment, the system includes logic for enabling the individual non-volatile storage columns. For example, the first non-volatile storage unit of each of the banks can be enabled or selected.
In one embodiment, the system includes logic for transferring portions of data from the interface buffers to the non-volatile storage columns at every interval of the input bus speed.
In one embodiment, a triple round-robin is used to transfer the data from the plurality of buffers to the non-volatile storage units. The outermost round-robin selects one of the columns in each set. The middle round-robin selects among the entries of the page size of the input buffer in the non-volatile storage units. The innermost round-robin selects one of the banks in the plurality of banks in a round-robin fashion. Then data is transferred from the selected buffer to the selected column.
In one embodiment, the burst data is received in 16,384 data portions each the width of the input bus of, for example, 64 bits per portion.